Imitation learning (IL) is a simple and powerful way to use high-quality human driving data, which can be collected at scale, to identify driving preferences and produce human-like behavior. However, policies based on imitation learning alone often fail to sufficiently account for safety and reliability concerns. In this paper, we show how imitation learning combined with reinforcement learning using simple rewards can substantially improve the safety and reliability of driving policies over those learned from imitation alone. In particular, we use a combination of imitation and reinforcement learning to train a policy on over 100k miles of urban driving data, and measure its effectiveness in test scenarios grouped by different levels of collision risk. To our knowledge, this is the first application of a combined imitation and reinforcement learning approach in autonomous driving that utilizes large amounts of real-world human driving data.

We propose JFP, a Joint Future Prediction model that can learn to generate accurate and consistent multi-agent future trajectories. For this task, many different methods have been proposed to capture social interactions in the encoding part of the model, however, considerably less focus has been placed on representing interactions in the decoder and output stages. As a result, the predicted trajectories are not necessarily consistent with each other, and often result in unrealistic trajectory overlaps. In contrast, we propose an end-to-end trainable model that learns directly the interaction between pairs of agents in a structured, graphical model formulation in order to generate consistent future trajectories. It sets new state-of-the-art results on Waymo Open Motion Dataset (WOMD) for the interactive setting. We also investigate a more complex multi-agent setting for both WOMD and a larger internal dataset, where our approach improves significantly on the trajectory overlap metrics while obtaining on-par or better performance on single-agent trajectory metrics.

Autonomous driving is an exciting new industry, posing important research questions. Within the perception module, 3D human pose estimation is an emerging technology, which can enable the autonomous vehicle to perceive and understand the subtle and complex behaviors of pedestrians. While hardware systems and sensors have dramatically improved over the decades -- with cars potentially boasting complex LiDAR and vision systems and with a growing expansion of the available body of dedicated datasets for this newly available information -- not much work has been done to harness these novel signals for the core problem of 3D human pose estimation. Our method, which we coin HUM3DIL (HUMan 3D from Images and LiDAR), efficiently makes use of these complementary signals, in a semi-supervised fashion and outperforms existing methods with a large margin. It is a fast and compact model for onboard deployment. Specifically, we embed LiDAR points into pixel-aligned multi-modal features, which we pass through a sequence of Transformer refinement stages. Quantitative experiments on the Waymo Open Dataset support these claims, where we achieve state-of-the-art results on the task of 3D pose estimation.

2D-to-3D reconstruction is an ill-posed problem, yet humans are good at solving this problem due to their prior knowledge of the 3D world developed over years. Driven by this observation, we propose NeRDi, a single-view NeRF synthesis framework with general image priors from 2D diffusion models. Formulating single-view reconstruction as an image-conditioned 3D generation problem, we optimize the NeRF representations by minimizing a diffusion loss on its arbitrary view renderings with a pretrained image diffusion model under the input-view constraint. We leverage off-the-shelf vision-language models and introduce a two-section language guidance as conditioning inputs to the diffusion model. This is essentially helpful for improving multiview content coherence as it narrows down the general image prior conditioned on the semantic and visual features of the single-view input image. Additionally, we introduce a geometric loss based on estimated depth maps to regularize the underlying 3D geometry of the NeRF. Experimental results on the DTU MVS dataset show that our method can synthesize novel views with higher quality even compared to existing methods trained on this dataset. We also demonstrate our generalizability in zero-shot NeRF synthesis for in-the-wild images.

流行的对象检测度量平均精度（3D AP）依赖于预测的边界框和地面真相边界框之间的结合。但是，基于摄像机的深度估计的精度有限，这可能会导致其他合理的预测，这些预测遭受了如此纵向定位错误，被视为假阳性和假阴性。因此，我们提出了流行的3D AP指标的变体，这些变体旨在在深度估计误差方面更具允许性。具体而言，我们新颖的纵向误差耐受度指标，Let-3D-AP和Let-3D-APL，允许预测的边界框的纵向定位误差，最高为给定的公差。所提出的指标已在Waymo Open DataSet 3D摄像头仅检测挑战中使用。我们认为，它们将通过提供更有信息的性能信号来促进仅相机3D检测领域的进步。

Panoptic图像分割是计算机视觉任务，即在图像中查找像素组并为其分配语义类别和对象实例标识符。由于其在机器人技术和自动驾驶中的关键应用，图像细分的研究变得越来越流行。因此，研究社区依靠公开可用的基准数据集来推进计算机视觉中的最新技术。但是，由于将图像标记为高昂的成本，因此缺乏适合全景分割的公开地面真相标签。高标签成本还使得将现有数据集扩展到视频域和多相机设置是一项挑战。因此，我们介绍了Waymo Open DataSet：全景视频全景分割数据集，这是一个大型数据集，它提供了用于自主驾驶的高质量的全景分割标签。我们使用公开的Waymo打开数据集生成数据集，利用各种相机图像集。随着时间的推移，我们的标签是一致的，用于视频处理，并且在车辆上安装的多个摄像头保持一致，以了解全景的理解。具体而言，我们为28个语义类别和2,860个时间序列提供标签，这些标签由在三个不同地理位置驾驶的自动驾驶汽车上安装的五个摄像机捕获，从而导致总共标记为100k标记的相机图像。据我们所知，这使我们的数据集比现有的数据集大量数据集大的数量级。我们进一步提出了一个新的基准，用于全景视频全景分割，并根据DeepLab模型家族建立许多强大的基准。我们将公开制作基准和代码。在https://waymo.com/open上找到数据集。

由于其在自主驾驶中的应用，因此基于单眼图像的3D感知已成为一个活跃的研究领域。与基于激光雷达的技术相比，单眼3D感知（包括检测和跟踪）的方法通常会产生较低的性能。通过系统的分析，我们确定了每个对象深度估计精度是界限性能的主要因素。在这种观察过程中，我们提出了一种多级融合方法，该方法将不同的表示（RGB和伪LIDAR）和跨多个对象（Tracklets）的时间信息结合在一起，以增强对目标深度估计。我们提出的融合方法实现了Waymo打开数据集，KITTI检测数据集和Kitti MOT数据集的每个对象深度估计的最新性能。我们进一步证明，通过简单地用融合增强的深度替换估计的深度，我们可以在单眼3D感知任务（包括检测和跟踪）方面取得重大改进。

3D对象检测是安全关键型机器人应用（如自主驾驶）的关键模块。对于这些应用，我们最关心检测如何影响自我代理人的行为和安全性（Egocentric观点）。直观地，当它更有可能干扰自我代理商的运动轨迹时，我们寻求更准确的对象几何描述。然而，基于箱交叉口（IOU）的电流检测指标是以对象为中心的，并且不设计用于捕获物体和自助代理之间的时空关系。为了解决这个问题，我们提出了一种新的EnoCentric测量来评估3D对象检测，即支持距离误差（SDE）。我们基于SDE的分析显示，EPECENTIC检测质量由边界框的粗糙几何形状界定。鉴于SDE将从更准确的几何描述中受益的洞察力，我们建议将物体代表为Amodal轮廓，特别是Amodal星形多边形，并设计简单的模型，椋鸟，预测这种轮廓。我们对大型Waymo公开数据集的实验表明，与IOU相比，SDE更好地反映了检测质量对自我代理人安全的影响;恒星的估计轮廓始终如一地改善最近的3D对象探测器的Enocentric检测质量。

预测道路用户的未来行为是自主驾驶中最具挑战性和最重要的问题之一。应用深度学习对此问题需要以丰富的感知信号和地图信息的形式融合异构世界状态，并在可能的期货上推断出高度多模态分布。在本文中，我们呈现MultiPath ++，这是一个未来的预测模型，实现了在流行的基准上实现最先进的性能。 MultiPath ++通过重新访问许多设计选择来改善多径架构。第一关键设计差异是偏离基于图像的基于输入世界状态的偏离，有利于异构场景元素的稀疏编码：多径++消耗紧凑且有效的折线，直接描述道路特征和原始代理状态信息（例如，位置，速度，加速）。我们提出了一种背景感知这些元素的融合，并开发可重用的多上下文选通融合组件。其次，我们重新考虑了预定义，静态锚点的选择，并开发了一种学习模型端到端的潜在锚嵌入的方法。最后，我们在其他ML域中探索合奏和输出聚合技术 - 常见的常见域 - 并为我们的概率多模式输出表示找到有效的变体。我们对这些设计选择进行了广泛的消融，并表明我们所提出的模型在协会运动预测竞争和Waymo开放数据集运动预测挑战上实现了最先进的性能。

Behavior prediction in dynamic, multi-agent systems is an important problem in the context of self-driving cars, due to the complex representations and interactions of road components, including moving agents (e.g. pedestrians and vehicles) and road context information (e.g. lanes, traffic lights). This paper introduces VectorNet, a hierarchical graph neural network that first exploits the spatial locality of individual road components represented by vectors and then models the high-order interactions among all components. In contrast to most recent approaches, which render trajectories of moving agents and road context information as bird-eye images and encode them with convolutional neural networks (ConvNets), our approach operates on a vector representation. By operating on the vectorized high definition (HD) maps and agent trajectories, we avoid lossy rendering and computationally intensive ConvNet encoding steps. To further boost VectorNet's capability in learning context features, we propose a novel auxiliary task to recover the randomly masked out map entities and agent trajectories based on their context. We evaluate VectorNet on our in-house behavior prediction benchmark and the recently released Argoverse forecasting dataset. Our method achieves on par or better performance than the competitive rendering approach on both benchmarks while saving over 70% of the model parameters with an order of magnitude reduction in FLOPs. It also outperforms the state of the art on the Argoverse dataset.